Kernel Fisher Discriminant Analysis in Gaussian Reproducing Kernel Hilbert Spaces –Theory

Kernel Fisher discriminant analysis (KFDA) has been proposed for nonlinear binary classification. It is a hybrid method of the classical Fisher linear discriminant analysis and a kernel machine. Experimental results have shown that the KFDA performs slightly better in terms of prediction error than the popular support vector machines and is a strong competitor to the latter. However, there is very limited statistical justification of this method. This article aims to provide a fundamental study for it in the framework of Gaussian reproducing kernel Hilbert spaces (RKHS). The success of KFDA is mainly due to two attractive features. (1) The KFDA has the flexibility of a nonparametric model using kernel mixture, (2) while its implementation algorithm uses a parametric notion via kernel machine (being linear in an RKHS and arising from the log-likelihood ratio of Gaussian measures). The KFDA emerging from the machine learning community can be linked to some classical results of discrimination of Gaussian processes in Hilbert space (Grenander, 1950). One of the main purposes of this article is to provide a justification of the underlying Gaussian assumption. It is shown, under suitable Running title: Kernel Fisher Discriminant Analysis in Gaussian RKHS. Corresponding author. 1 conditions, most low dimensional projections of kernel transformed data in an RKHS are approximately Gaussian. The study of approximate Gaussianity of kernel data is not limited to the KFDA context. It can be applied to general kernel machines as well, e.g., support vector machines.